Repository navigation
Conversation
`SparkHllSketch` and `SparkHllUnion` inferred their heap size from `estimate()`, and `SparkHllUnion::merge` called `estimate()` on both sides before every update to keep a `dense` flag for that inference. `datasketches` 0.6.0 exposes the size directly: `estimated_size()` reads the coupon or register array length in O(1) and is exact. Use it, and drop the `layout` inference, both `dense` flags and the `preamble` module. On 0.6.0 this makes Final-mode `hll_sketch_agg` and `hll_union_agg` over many tiny groups about 20% faster (211 to 169 ns per partial row at lgConfigK 18). It matters more once Comet moves to a datasketches release with apache/datasketches-rust#317, which defers the register cache rebuild after a dense merge until the next query. The per-merge `estimate()` forced that rebuild on every merge. Built against apache#317, merging an already dense sketch drops from 266 us to 5.5 us, close to the 5.2 us a plain register-wise max takes. Part of apache#7341.
comphead
force-pushed
the
hll-estimated-size-7341
branch
from
November 15, 2026 19:22
ef10fc2 to
ffa0694
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #7341. #7341 stays open for item 2 (a
GroupsAccumulator) and for thedatasketchesbump once a release includes apache/datasketches-rust#317.Rationale for this change
#7345 fixed the problem in the title of #7341. Item 3 is still real on
main: merging a partial sketch that is already dense costs about 270 µs at lgConfigK 18, while the register-wise max it needs takes 5 µs.datasketches0.6.0 rebuilds its register cache, onef64division per register, after every dense merge.apache/datasketches-rust#317 (merged 2026-11-04, not released yet) defers that rebuild until the next query. Comet would not benefit from it.
SparkHllUnion::mergecallsestimate()on the union and on the incoming sketch before every update, to keep adenseflag for the heap-size inference, andheap_size()callsestimate()on every accumulatorsize(). Each of those calls would force the deferred rebuild.datasketches0.6.0 exposes the size directly.HllSketch::estimated_size()andHllUnion::estimated_size()read the coupon array or register array length in O(1), and the result is exact. The estimate-based inference predates that API, since 0.3.0 kept the layout private.What changes are included in this PR?
SparkHllSketch::heap_size()andSparkHllUnion::heap_size()return the crate'sestimated_size().layoutmodule, bothdenseflags and thepreamblemodule are removed, somergeandsize()no longer compute estimates.heap_size_follows_the_crates_layoutnow asserts thatheap_size()equals the allocation at every step, instead of "never below, at most one size step above".How are these changes tested?
The existing heap-size tests cover it.
heap_size_follows_the_crates_layoutnow requires an exact match with the allocation the crate serializes at every cardinality, through each SET resize and the promotion.high_lg_config_k_is_charged_for_what_it_holds,union_heap_size_follows_the_crates_layoutandsingleton_groups_at_high_lg_config_k_fit_a_small_poolare unchanged. Sketch results do not change: only the size accounting is touched, and the probe below gave identical output bytes in every build.The numbers come from a throwaway example (not in this PR) that runs a Final
AggregateExecover partial states at lgConfigK 18, with rows interleaved one partial per group per batch, the way a reducer reads map outputs. Release build on an Apple M-series laptop,hll_sketch_agg, ns per partial row, median of 5 runs over two interleaved rounds.hll_union_agggives the same picture.With
datasketches0.6.0, as shipped by this PR:mainWith
datasketchesat apache/datasketches-rust@5f080f98e5 (#317, still version 0.6.0, through[patch.crates-io], not part of this PR):mainSparkHllUnion::mergeof a dense sketchA plain 2^18 register-wise max takes 5.2 µs, so with #317 a dense merge costs about what the merge itself has to do. Differences under 5% are within run-to-run noise here.